Add script to build migrations with datastore plugin #126

gammazero · 2021-03-20T05:06:32Z

Update documentation

Stebalien · 2021-04-12T17:51:01Z

aschmahmann · 2021-04-12T18:36:09Z

@gammazero do you mind rebasing this on master and then we can do a walkthrough on how this works and make sure the script is mergeable?

gammazero · 2021-04-14T19:14:50Z

I needed to update the go-ipfs dependency for fs-repo-10-to-11 because the plugin build script gets the go-ipfs dependency from the migration's go.mod file and the version that fs-repo-10-to-11 previously had was not found in github.

aschmahmann · 2021-04-14T19:52:56Z

@gammazero hmmm... I see we accidentally did a force push ipfs/kubo#7857 (comment). I restored the branch but we should probably update anyway.

If the script works now that I've restored the old branch then let's drop the update from this PR and just do it in another one ideally once the next go-ipfs release is out. Otherwise, we can make a PR that just updates the deps now and have this PR build off of that one.

I want this PR to be two files instead of 700 😄

gammazero · 2021-04-15T00:03:50Z

@aschmahmann After restoring the old branch, the script works. Now the PR has only one change.

The distribution at /ipfs/QmaaN2kipZfUpRSzwvUeG4Xi3yp1JJB294Vj8pnZ24hesF is no longer available and this is causing a sharness test to fail. Updated the test to use the current distribution.

aschmahmann · 2021-04-15T17:16:36Z

build-plugin.sh

@@ -0,0 +1,118 @@
+#!/bin/bash
+#
+# Building migrations with datastore plugin


How much of this is datastore specific, or just arbitrary Go plugins (e.g. could IPLD plugins work too)?

I see there are some implied datastore plugin names, but the plugin names could be separately specified. Is there anything else?

I assumed this was only for datastore plugins, but I guess it could be built with any. This is just for building a filestore repo migration, so does it even make sense to support any plugin? I do not know.

Do I need to add another argument, or prompt, for the name of the plugin?

aschmahmann · 2021-04-15T17:25:47Z

build-plugin.sh

+
+function usage() {
+    echo "usage: $0 plugin_repo x-to-y ...">&2
+    echo "example: $0 github.com/ipfs/go-ds-s3 10-to-11" >&2


Two feature requests I'd add here, that can happen after we merge this initial one but would be nice to have are:

Being able to choose the version of the repo not just the repo itself (e.g. I might need to build an old migration against an old version)

Being able to add multiple plugins to a given binary

A user can now specify the version of the migration by following it with @<version_or_hash>

The script builds a single migration, with one or more plugins.

Example:

./build-plugin.sh 10-to-11 github.com/ipfs/go-ds-s3 github.com/ipfs/go-ds-swift@v0.1.0

Awesome, looking forward to giving this a spin.

Script now builds one migration, but with any number of specified plugins. Example `./build-plugin.sh 10-to-11 github.com/ipfs/go-ds-s3 github.com/ipfs/go-ds-swift@v0.1.0`

aschmahmann

Does the job for the repos tested, but makes some assumptions about the plugins system we should spell out.

Also, let's add a note in the README that this script exists and might be very useful to people who are running into problems migrating their go-ipfs nodes that use plugins.

aschmahmann · 2021-04-16T21:25:24Z

build-plugin.sh

+    local plugin_name="$(echo $plugin_repo | rev | cut -d '-' -f 1 | rev)"
+    local ds_name="${plugin_name}ds"


I'm pretty sure that the name of the plugin as added to the preload_list is only used for populating the import directives in the code generation script https://github.com/ipfs/go-ipfs/blob/master/plugin/loader/preload.sh.

It'd be nice if the user could set this in case they somehow end up with a conflict, but I don't think it should matter/come up for almost anyone.

Done.

When a plugin name conflict is detected, suggest a new name by appending a "1" to the current name. If running interactive, then ask the user to use the new name or provide one. If running non-interactive, then keep appending "1" until no conflict.

aschmahmann · 2021-04-16T21:31:49Z

build-plugin.sh

+    pushd "$BUILD_GOIPFS"
+    go get "${plugin_repo}@${plugin_version}"
+    popd
+    echo "$ds_name ${plugin_repo}/plugin 0" >> "${BUILD_GOIPFS}/plugin/loader/preload_list"


${plugin_repo}/plugin 0

This part is incorrect/needs to be configured by the user. The actual subpackage that the plugin is in is totally configurable by the user and so should be something passed in by the user. This doesn't seem like it's too bad since we can just rip it off of the URL that's passed in.

The 0 part is hopefully fine for almost everyone, but technically this can either be any number or '*' and the user should be able to configure it. This is maybe more annoying to pass in, if we needed to default to something maybe choose * since it's what we use in our preload examples https://github.com/ipfs/go-ipfs/blob/0ae9b2b9034a57e02df419fe72f705f5a5b3b9c7/plugin/loader/preload_list.

Since this needs to be specified for each plugin that is build, I think it makes for a much less messy command line if the user is asked interactively about which plugin to load. Now, for each plugin repo build, the user gets a prompt that looks like:

For github.com/ipfs/go-ds-s3, load 'plugin *' [y/n]?

If they answer n, then they are asked to for a different value.

The prompting can be avoided by using the -y flag as the first argument to the scipt.

When a plugin name conflict is detected, suggest a new name by appending a "1" to the current name. If running interactive, then ask the user to use the new name or provide one. If running non-interactive, then keep appending "1" until no conflict.

- README describes how to use the build-plugin.sh script to build a migration with plugins. - Display help with `build-plugin.sh -h`

gammazero · 2021-04-21T22:56:39Z

@aschmahmann Added section to README to describe build-plugin.sh script and how to use it. Script also displays help when given -h flag.

gammazero requested a review from aschmahmann March 20, 2021 05:06

Base automatically changed from feat/issue-98 to master April 5, 2021 20:12

gammazero force-pushed the feat/build-with-plugin branch from 0344c82 to b396d4b Compare April 14, 2021 19:10

Add script to build migration with plugin

7452cb4

gammazero force-pushed the feat/build-with-plugin branch from b396d4b to 7452cb4 Compare April 15, 2021 00:02

Update test to use a distribution that is available

9630e23

The distribution at /ipfs/QmaaN2kipZfUpRSzwvUeG4Xi3yp1JJB294Vj8pnZ24hesF is no longer available and this is causing a sharness test to fail. Updated the test to use the current distribution.

aschmahmann reviewed Apr 15, 2021

View reviewed changes

gammazero added 3 commits April 16, 2021 00:52

Allow user to specify version of plugin repo to build

448beef

Build a migration with one or more plugins

fea3ae2

Script now builds one migration, but with any number of specified plugins. Example `./build-plugin.sh 10-to-11 github.com/ipfs/go-ds-s3 github.com/ipfs/go-ds-swift@v0.1.0`

Only build go-ipfs once, even with multiple plugins

5b095e5

aschmahmann requested changes Apr 16, 2021

View reviewed changes

gammazero added 2 commits April 16, 2021 14:47

Specify loading all sub-plugins

fee094f

prompt user for plugin to load

586c24d

gammazero requested a review from aschmahmann April 20, 2021 18:32

gammazero added 2 commits April 21, 2021 15:05

Udate documentation to tell about build-plugin.sh

b9fb061

- README describes how to use the build-plugin.sh script to build a migration with plugins. - Display help with `build-plugin.sh -h`

aschmahmann approved these changes Jun 17, 2021

View reviewed changes

aschmahmann merged commit ea2bc1a into master Jun 17, 2021

gammazero deleted the feat/build-with-plugin branch July 3, 2021 06:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add script to build migrations with datastore plugin #126

Add script to build migrations with datastore plugin #126

gammazero commented Mar 20, 2021

Stebalien commented Apr 12, 2021

aschmahmann commented Apr 12, 2021

gammazero commented Apr 14, 2021

aschmahmann commented Apr 14, 2021

gammazero commented Apr 15, 2021

aschmahmann Apr 15, 2021 •

edited

Loading

gammazero Apr 20, 2021

aschmahmann Apr 15, 2021

gammazero Apr 16, 2021

aschmahmann Apr 16, 2021

aschmahmann left a comment

aschmahmann Apr 16, 2021

gammazero Apr 21, 2021

aschmahmann Apr 16, 2021

gammazero Apr 20, 2021 •

edited

Loading

gammazero commented Apr 21, 2021

		local plugin_name="$(echo $plugin_repo \| rev \| cut -d '-' -f 1 \| rev)"
		local ds_name="${plugin_name}ds"

Add script to build migrations with datastore plugin #126

Add script to build migrations with datastore plugin #126

Conversation

gammazero commented Mar 20, 2021

Stebalien commented Apr 12, 2021

aschmahmann commented Apr 12, 2021

gammazero commented Apr 14, 2021

aschmahmann commented Apr 14, 2021

gammazero commented Apr 15, 2021

aschmahmann Apr 15, 2021 • edited Loading

Choose a reason for hiding this comment

gammazero Apr 20, 2021

Choose a reason for hiding this comment

aschmahmann Apr 15, 2021

Choose a reason for hiding this comment

gammazero Apr 16, 2021

Choose a reason for hiding this comment

aschmahmann Apr 16, 2021

Choose a reason for hiding this comment

aschmahmann left a comment

Choose a reason for hiding this comment

aschmahmann Apr 16, 2021

Choose a reason for hiding this comment

gammazero Apr 21, 2021

Choose a reason for hiding this comment

aschmahmann Apr 16, 2021

Choose a reason for hiding this comment

gammazero Apr 20, 2021 • edited Loading

Choose a reason for hiding this comment

gammazero commented Apr 21, 2021

aschmahmann Apr 15, 2021 •

edited

Loading

gammazero Apr 20, 2021 •

edited

Loading